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DISA  Test,  Evaluation,  and  Certification:  A  New 
Organizational  Construct 
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Test,  evaluation,  and  certification  (TE&C)  is  an  essential  part  of  the  Department  of  Defense 
(DOD)  acquisition  system.  As  the  DOD  moves  forward  with  the  implementation  of  a  new 
acquisition  process  for  information  technologies,  the  Defense  Information  Systems  Agency  (DISA) 
must  lead  from  the  front.  The  DISA  TE&C  organizations  bring  tremendous  expertise  across  all 
information  technology  (IT)  TE&C  disciplines:  developmental  testing,  operational  testing,  joint 
interoperability  test  and  certification,  and  security  testing  or  information  assurance  certification. 
However,  we  must  evolve  beyond  our  traditional  methodologies  to  provide  the  same  rigorous  services 
in  the  agile  IT  environment.  This  article  describes  the  new  DISA  test  and  evaluation  organizational 
structure  and  the  goals  that  we  have  established  to  responsive,  mission-focused  TE&C  services  that 
enable  rapid  acquisition  of  enhanced  information  technologies  for  the  warfighter. 
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The  Defense  Information  Systems  Agen¬ 
cy  (DISA)  is  committed  to  advancing 
the  art  and  science  of  test,  evaluation, 
and  certification  (TE&C)  of  informa¬ 
tion  technology  (IT)  and  national  secu¬ 
rity  systems  (NSSs).  The  DISA  TE&C  mission  is  to 
provide  responsive,  agile,  and  cost-effective  interopera¬ 
bility,  other  tests,  evaluations,  and  certifications;  or  both 
to  support  rapid  acquisition  and  fielding  of  enhanced 
net-centric  warfighting  capabilities. 

Clearly,  TE&C  is  an  essential  element  of  the 
Department  of  Defense  (DOD)  acquisition  system. 
As  the  DOD  moves  forward  with  agile  IT  acquisition 
concepts,  DISA  must  be  at  the  forefront  of  shaping  the 
supporting  policies,  processes,  and  joint  TE&C 
environment.  DISA  TE&C  organizations  must  pro¬ 
gress  toward  the  use  of  distributed  test  methodologies 
that  provide  realism  and  promote  joint  mission 
effectiveness.  DISA’s  early  and  continuous  involve¬ 
ment  throughout  the  IT  acquisition  life  cycle  with 
service  proponents,  operational  sponsors,  and  industry 
developers  will  help  ensure  that  agility  is  achieved. 

DISA  TE&C  organization 

DISA  recently  restructured  its  testing  organization. 
The  new  structure  better  aligns  test  and  certification 
activities  with  the  agency’s  strategic  objectives.  DISA 


TE&C  organizational  elements  are  aligned  under  the 
DISA’s  test  and  evaluation  (T&E)  executive,  and 
include  the  Office  of  the  Test  and  Evaluation 
Executive  (TEO),  Joint  Interoperability  Test  Com¬ 
mand  0ITC),  and  the  Test  and  Evaluation  Manage¬ 
ment  Center  (TEMC).  DISA  TE&C  organizations 
and  facilities  are  located  in  Fort  Huachuca,  Arizona; 
Indian  Head,  Maryland;  and  Falls  Church,  Virginia. 
The  Falls  Church  organization  and  testing  labs  will 
move  to  Fort  Meade,  Maryland,  in  April  2011,  as  part 
of  the  Base  Realignment  and  Closure.  Figure  1  depicts 
the  overarching  DISA  TE&C  organizational  structure. 

Collectively,  DISA  TE&C  organizations  and  asso¬ 
ciated  facilities  serve  as  the  IT  test  bed  in  the  Major 
Range  and  Test  Facility  Base  (MRTFB)  and  provide 
capabilities  and  infrastructure  for  end-to-end  system 
engineering  and  execution  of  distributed,  net-centric 
testing  of  core  technologies  and  mission-enabling 
applications  supporting  the  joint  warfighter.  The 
DISA  MRTFB  is  a  national  asset  that  provides  full 
spectrum  TE&C  services  in  support  of  DOD,  other 
government  agencies,  and  industry. 

The  DISA  T&E  executive  provides  oversight  of  all 
DISA  T&E  activities.  The  TEO  is  responsible  for 
serving  in  advisory  and  support  roles  within  various 
Office  of  the  Secretary  of  Defense,  DISA,  and  joint 
governance  groups  ( Figure  2).  The  TEO  influences 
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Figure  1.  Defense  Information  Systems  Agency  Test, 
Evaluation,  and  Certification  organizational  structure. 


DOD  IT  TE&C  strategies,  policies,  procedures,  and 
investments.  The  TEO  organizational  structure  con¬ 
sists  of  the  deputy  T&E  executive,  administrative  staff, 
and  an  advisory  staff.  The  Interoperability  Policy  and 
Certification  Panel  (IP/ICP)  lead  serves  as  DISA’s 
voting  member  on  the  Military  Communications  and 
Electronics  Board  IP/ICP  and  represents  the  DISA 
T&E  executive  at  various  senior-level  Office  of  the 
Secretary  of  Defense  T&E  working  groups  and 
advisory  boards.  The  Joint  Test  &  Evaluation  Meth¬ 
odology  Transition  director  works  in  cooperation  with 
the  operational  T&E  director  to  improve  the  ability  to 
conduct  testing  in  a  joint  environment  and  validates 
the  capability  test  methodology  to  define  and  use  a 
distributed,  live,  virtual,  constructive  test  environment 
to  evaluate  joint  mission  effectiveness.  The  Joint  Test  & 
Evaluation  (JT&E)  liaison  supports  the  JT&E  Program 
and  serves  as  a  conduit  to  DISA  for  all  chartered  and 
proposed  JT&E  efforts  and  quick  reaction  tests.  Lastly, 
the  strategic  planning  and  communications  chief  serves 
as  the  principal  adviser  for  defining  organizational 
planning  and  outreach  strategies  associated  with  all 
TE&C  elements  within  DISA. 


Joint  interoperability  test  command 

As  designated  by  the  Joint  Chiefs  of  Staff,  JITC 
certifies  IT  and  NSS  interoperability  and  net-readiness 
for  joint  military  operations  ( Figure  J).  In  addition  to 


serving  as  an  operational  test  agency,  JITC  serves  as 
the  responsible  test  organization  for  various  DOD 
program  offices.  JITC  works  closely  with  the  warfight¬ 
ing  combatant  commanders  during  exercises  and 
contingency  operations,  providing  them  on-the-spot 
evaluations  of  problem  areas  and  viable  mission- 
oriented  solutions.  The  laboratories  at  JITC  operate 
as  an  MRTFB.  JITC’s  global  reach  extends  to  the 
entire  spectrum  of  DOD,  federal  government,  private 
industry,  and  allies  in  support  of  command  and  control, 
intelligence,  and  defense  reform  initiatives.  JITC  deals 
directly  with  vendors  to  provide  critical  preacquisition 
test  results.  This  early  involvement  in  development 
results  in  better  systems  at  lower  cost. 

JITC’s  mission  is  supported  by  a  blend  of  military, 
civilian,  and  contractor  personnel,  including  engineers, 
computer  scientists,  and  technical  and  operational 
experts.  JITC  government  personnel  provide  technical 
direction,  policy  decisions,  schedules,  and  program  cost 
controls  in  the  management  of  JITC  daily  operations. 
The  organization  is  primarily  composed  of  divisions  and 
portfolios  and  is  aligned  with  the  warfighting  mission 
areas  to  provide  consistent  practices  and  processes  across 
the  testing  disciplines  (developmental,  operational, 
interoperability,  and  security)  and  support  the  imple¬ 
mentation  of  a  risk-based  test  strategy  that  enables  agile 
testing  and  rapid  fielding.  Divisions  offer  business  and 
internal  services  to  JITC  to  perform  its  mission,  and  the 
portfolios  execute  the  JITC  mission  and  address 
customer  requirements.  JITC  elements  are  as  follows: 

•  The  Operational  Test  &  Evaluation  Division 
conducts  operational  testing  of  IT  and  NSSs 
acquired  by  DISA  and  other  DOD  organizations 
to  ensure  capabilities  are  effective,  suitable, 
interoperable,  and  secure.  This  division  assists 
in  the  preparation  of  critical  operational  issues 
and  develops,  defines,  and  publishes  measures  of 
effectiveness,  measures  of  suitability,  and  mea¬ 
sures  of  performance.  It  also  directs  and  approves 
operational  T&E  methods  for  data  collection, 
reduction,  and  analysis. 

•  The  Business  Management  Division  provides 
management  and  oversight  of  all  JITC  business, 
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Figure  2.  Office  of  the  Test  and  Evaluation  Executive  organizational  structure. 
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personnel,  training,  contracts,  and  security  oper¬ 
ations  in  support  of  JITC’s  mission,  goals,  and 
objectives  while  obtaining  or  providing  best-value 
goods  and  services  for  JITC’s  workforce.  This 
includes  command  support  activities,  corporate- 
level  management  of  proposals,  and  contracts  or 
agreements.  This  division  also  serves  as  the 
MRTFB  focal  point. 

•  The  Warfighter  Support  Division  assists  the  JITC 
commander  with  setting  command  priorities  and 
moderating  the  operational  tempo  in  support  of 
day-to-day  command  operations.  It  provides 
exercise  and  contingency  support  to  the  warfight¬ 
er  and  combatant  commander  by  participation  in 
combatant  commander  exercises,  deploying  to 
contingencies,  and  providing  a  24/7  technical 
support  hotline. 

•  The  Strategic  Planning  &  Engineering  Division 
provides  leading  edge  test  engineering,  instru¬ 
mentation,  and  operations  services  to  the  JITC 
divisions  and  portfolios  in  support  of  the  overall 
JITC  mission.  This  support  includes  defining  and 
coordinating  the  integration  of  test  engineering 
best  practices,  oversight  of  key  acquisition  test 
programs,  and  establishing  standard  practices  for 
instrumentation  development. 

•  The  Test  Bed  Operations ,  Network  &  Infrastructure 
Division  plans  for  and  maintains  functional  test 
beds,  local  and  wide  area  networks,  and  labora¬ 


tories  at  JITC.  It  provides  timely  facilities 
infrastructure  and  logistics  support  services  reso¬ 
lution  to  JITC’s  test  portfolios  and  divisions. 

•  The  Enterprise  Services  Portfolio  supports  the 
fielding  of  global  net-centric  solutions  by  pro¬ 
viding  continuous  and  effective  T&E  services  to 
DISA  and  DOD  joint  acquisition  programs 
within  the  enterprise  services  construct. 

•  The  Focused  Logistics  and  Business  Portfolio  serves 
as  the  focused  logistics  and  business  mission  area 
responsible  test  organization  for  the  DOD  and 
other  federal  agencies.  It  also  assists  programs  in 
the  transition  to  a  capability-based  net-centric 
environment. 

•  The  Force  Application  or  Force  Protection  Por  folio 
conducts  interoperability  assessments,  standards 
conformance,  and  interoperability  certification 
testing  of  force  application  or  protection  systems 
and  joint  tactical  data  links  in  hardware-in-the- 
loop  and  operationally  realistic  environments  to 
validate  the  implementation  of  approved  stan¬ 
dards  and  certify  system  interoperability  in  a  joint 
environment. 

•  The  Command  and  Control  ( C2)  and  Battle  space 
Awareness  (BA)  Por  folio  mitigates  risk  to  C2  and 
BA  community  programs  and  activities  by 
providing  developmental  and  interoperability 
T&E  support  and  certification  services  with  the 
goal  of  enhancing  C2  and  BA  systems  capabilities 
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Figure  4.  Test  and  Evaluation  Management  Center  organizational  structure. 


and  performance  in  support  of  joint  and  coalition 
warfighters. 

•  The  Battlespace  Communications  Portfolio  supports 
the  warfighter  with  direct  technical  assistance  and 
through  testing  of  secure  and  nonsecure  voice, 
video,  and  data  communications.  It  characterizes 
performance  and  mitigates  risk  as  net-centric 
transport  solutions  are  introduced  to  the  global 
information  grid. 

•  The  National  Intelligence  Portfolio  mitigates  risk  to 
national  intelligence  community  programs  and 
activities  by  providing  support  through  develop¬ 
mental  and  interoperability  T&E  and  certification 
efforts  to  enhance  national  collection  or  dissem¬ 
ination  systems  capabilities  and  performance. 

•  The  Homeland  Security  or  Information  Assurance 
Portfolio  provides  the  full  range  of  testing  services, 
technical  support,  coordination,  and  oversight  to 
the  DOD,  Department  of  Homeland  Security, 
and  other  federal,  state,  and  local  agencies  to 
ensure  seamless  acquisition,  integration,  and 
Information  Assurance  (IA)  of  systems  supporting 
the  DOD  and  the  National  Command  Authority. 

Test  and  Evaluation  Management  Center 

The  TEMC  is  a  staff  organization  responsible  for 
strategic  management  of  agency  test  resources.  TEMC 
provides  TE&C  oversight  and  guidance  to  DISA 
acquisition  programs  to  ensure  consistent  application 
of  sound  T&E  methodologies  and  processes.  TEMC 
has  highly  skilled  and  motivated  computer  scientists, 
electrical  engineers,  operations  research  analysts,  IT 
specialists,  and  management  analysts  who  establish, 
review,  and  enforce  TE&C  strategies,  policies,  and 
procedures  for  DISA  acquisition  programs.  They 


represent  DISA,  as  well  as  champion  strategies  within 
the  TE&C  community. 

The  TEMC  Strategic  Initiatives  &  Management 
Support  Branch  is  responsible  for  TEMC  resource 
management  in  the  areas  of  personnel,  finance,  and 
contracting  and  supports  the  daily  operations  mission 
by  executing  all  aspects  of  logistics,  training,  facilities, 
security,  base  realignment  and  closure  management, 
and  internal  controls.  The  Program  T&E  Support 
Branch  provides  guidance  in  the  implementation  of 
agile  T&E  strategies  and  methodologies  and  supports 
the  DISA  campaign  plan  by  formalizing  procedures  to 
provide  TE&C  services  to  the  DISA  development 
community  at  large  and  assist  in  planning  test  events 
necessary  to  provide  the  information  and  confidence 
that  DISA  program  capabilities  are  ready  for  imple¬ 
mentation.  The  Capabilities  Integration  Lab  Division 
operates  and  maintains  the  DISA  test  infrastructure, 
providing  on-demand  test  suites,  operational  network 
connectivity,  collaborative  environment,  multiple  se¬ 
curity  levels,  and  technical  support  services.  It  works  to 
ensure  program  system  integration  through  implemen¬ 
tation  of  seamless  integration  of  software  capabilities 
within  required  security  framework  and  leveraging 
operational  network  infrastructure  ( Figure  4). 

The  TEMC  laboratory  is  also  an  element  of  the 
MRTFB.  With  DISA  laboratories  scheduled  to  move 
to  Fort  Meade  beginning  in  2011,  the  TEMC  is 
dedicated  to  making  the  relocation  as  transparent  as 
possible  by  minimizing  disruption  to  services  and 
programs. 

DISA  campaign  plan 

In  January  2010,  Lieutenant  General  Carroll  F. 
Pollett  (DISA  director)  formally  established  a  cam- 
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paign  plan  to  focus  on  the  nation’s  need  for  a  force  that 
is  ready  and  capable  of  performing  the  full  range  of 
military  operations.  The  DISA  campaign  plan  defines 
the  major  focus  areas  and  goals  that  will  enable  DISA 
to  meet  the  demands  of  the  warfighter.  The  campaign 
plan  is  centered  on  “lines  of  operation”  and  supported 
by  “joint  enablers”  that  provide  the  framework  for 
strategic  planning,  budgeting,  and  task  prioritization  or 
execution.  The  plan  lays  out  the  methods  by  which 
DISA  will  satisfy  its  goals  and  achieve  its  objectives. 

Within  the  campaign  plan,  the  three  lines  of 
operation  are  as  follows: 

•  enterprise  infrastructure, 

•  C2  and  information  sharing, 

•  operate  and  assure. 

The  joint  enablers  are  the  process  and  governance 
areas  that  support  the  lines  of  operation.  The  nine  joint 
enablers  are  as  follows: 

•  acquisition, 

•  contracting, 

•  engineering, 

•  information  and  knowledge  management, 

•  people, 

•  planning, 

•  resources, 

•  spectrum, 

•  testing. 

The  DISA  TE&C  strategic  plan  coincides  with  the 
overarching  DISA  campaign  plan  joint  enabler  for 
testing,  which  outlines  the  priorities  in  clearly  stated  2- 
and  4-year  actions  for  accomplishing  the  TE&C 
mission. 

DISA  TE&C  objectives  and  goals 

The  primary  DISA  TE&C  objectives  are  as  follows: 

•  ensuring  mission-focused  agility  through  rapid 
development  of  mission-oriented  test  plans  that 
permit  objective  assessments  of  technical  and 
operational  capabilities  and  limitations; 

•  improving  TE&C  by  ensuring  integration  and 
synchronization  of  efforts  among  proponents, 
operational  sponsors,  developers,  and  testers; 

•  ensuring  use  of  consistent,  sound,  repeatable 
TE&C  strategies  that  can  be  executed  at  all 
levels,  by  any  test  organization,  and  that,  when 
executed,  yield  similar  results. 

Early  program  involvement  and  enhanced  methods 
or  processes  for  executing  TE&C  will  lead  to  rapid 
deployment  of  IT  and  NSS  capabilities  that  are 
operationally  effective,  suitable,  interoperable,  and 


secure.  To  improve  TE&C  processes,  DISA  has 
established  three  aggressive  goals: 

1.  provide  efficient,  responsive  joint  interoperability 
TE&C  and  other  capabilities  as  a  service; 

2.  establish  an  on-demand  TE&C  environment 
that  provides  enhanced  virtualization  and  access 
to  federated  capabilities  that  serve  the  TE&C 
community; 

3.  develop  and  retain  a  highly  qualified  and 
professional  workforce  to  ensure  the  success  of 
agile  TE&C  activities. 

To  achieve  these  goals  and  objectives,  we  must 
introduce  new  strategies  that  allow  us  to  adapt  to 
rapidly  changing  technologies,  ensure  that  we  stay 
engaged  with  the  program  offices  to  help  them  find 
and  fix  problems  early,  stay  objective  in  our  assess¬ 
ments,  and  make  the  right  investment  decisions  to 
keep  us  relevant  in  the  on-demand  world  of  IT. 
Strategies  for  satisfying  DISA  TE&C  goals  include 
the  following: 

•  enhance  the  execution  of  the  IT  and  NSS  TE&C 
mission  in  accordance  with  DOD  policy  and 
federal  law; 

•  enhance  the  execution  of  DISA’s  operational  test 
mission; 

•  reduce  TE&C  timelines  in  support  of  agile 
acquisition  through  implementation  of  the  capa¬ 
bility  TE&C  model; 

•  enhance  the  ability  to  execute  security  (IA) 

TE&C; 

•  work  with  capability  portfolio  managers  to  ensure 
a  system  of  system  testing  in  a  joint  operational 
context; 

•  implement  a  network  pilot  that  supports  devel¬ 
opment,  integration,  and  TE&C; 

•  improve  infrastructure  and  federate  capabilities 
across  DOD; 

•  implement  a  virtual  environment  in  support  of 

TE&C; 

•  maintain  a  focus  on  recruiting,  hiring,  and 
retaining  a  workforce  of  trained,  experienced 
TE&C  professionals. 

Tasks  and  initiatives  associated  with  respective 
DISA  TE&C  strategies  are  assigned  and  tracked  for 
accomplishment  to  verify  whether  goals  have  been  met 
and  objectives  have  been  satisfied  within  established 
timelines.  Several  key  initiatives  will  allow  DISA  to 
satisfy  their  strategic  TE&C  goals  and  objectives. 

For  example,  DISA  TE&C  organizations  will 
support  IT  acquisition  reform  in  accordance  with 
National  Defense  Authorization  Act  Section  804  by 
establishing  new  test  approaches  that  merge  TE&C 
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events  (i.e.,  development  testing,  operational  testing, 
net-ready  key  performance  parameter  validation,  and 
IA)  more  concurrently. 

Initiatives 

DISA  currently  provides  a  software  development 
environment  known  as  Forge.mil  (see  www.forge.mil). 

For  TE&C,  we  envision  TestForge.mil  as  a  virtual 
environment  that  enables  developers,  users,  and  testers 
to  rapidly  verify  new  software  that  satisfies  user  needs. 
The  environment  promotes  collaboration  among  key 
stakeholders,  synchronization  of  development  and 
testing,  and  integration  of  team  members,  and 
combined  execution  to  satisfy  the  decision-making 
needs  of  all  test  customers.  TestForge  provides  on- 
demand  access  to  testing  capabilities  throughout  the 
development  life  cycle  and  enables  capabilities  that 
support  continuous  integration  and  service  virtualiza¬ 
tion.  TestForge  currently  provides  support  for  testing 
within  the  development  cycle  on  agile  projects.  This 
includes  support  for  defect  management,  automated 
unit,  functional  and  regression  testing,  and  static  code 
analysis  through  a  continuous  integration  environment. 
In  the  near  future,  TestForge  will  provide  full  support 
for  all  testing  activities  including  performance,  seal- 
ability,  reliability,  interoperability,  operational,  net- 
ready  key  performance  parameters,  and  IA.  TestForge 
will  be  a  key  element  of  an  enterprise  test  and 
integration  capability  that  provides  access  to  common 
tools,  methodologies,  and  support. 

The  IA  Range  is  an  infrastructural  platform 
designed  to  integrate  distributed  and  heterogeneous 
IA  architectural  systems  and  solutions  and  the  DOD 
Computer  Network  Defense  (CND)  operational 
hierarchy  to  provide  DOD  stakeholders  with  a  venue 
to  strengthen  the  global  information  grid  security 
posture  by  supporting  operational  exercises,  training 
network  defenders,  and  testing  and  evaluating  new 
cyber  capabilities.  DISA  Field  Security  Operations 
manages  the  IA  Range  and  ensures  that  it  provides  a 
realistic  Internet  environment  and  tools  in  which  to 
test  application  security,  as  well  as  train  personnel  in 
computer  network  operations.  Field  Security  Opera¬ 
tions  intends  for  the  IA  Range  to  allow  for  network 
services  found  at  the  NetOps  tiers  1-3  and  provide  a 
joint  service  environment  for  cyber  exercises  and  CND 
AP  training.  In  support  of  its  DISA  TE&C  mission, 
the  IA  Range  incorporates  in-depth  defense  design 
principles  to  provide  DOD  organizations  with  a 


methodical,  repeatable,  and  verifiable  cyber  T&E 
framework  (supported  by  performance-based  metrics 
indicators)  designed  to  measure  (quantify  and  qualify) 
the  capabilities  and  limitations  of  network  defenders  to 
synergistically  integrate  people,  operations,  and  technol¬ 
ogy  to  protect,  monitor,  detect,  analyze  and  diagnose,  and 
respond  (contain,  eradicate,  and  recover)  to  cyber 
security  attacks.  DISA  TE&C  facilities  will  use  the 
IA  Range  to  promote  a  consistent,  repeatable,  and 
verifiable  T&E  venue  by  which  I A  and  CND  technical 
and  operational  concepts  will  be  validated  against 
requirements  and  specifications  for  improvement. 

To  further  professionalize  the  DISA  TE&C  work¬ 
force,  the  DISA  T&E  executive  sponsors  several  events 
and  efforts.  The  DISA  T&E  forums  are  quarterly 
professional  development  events  and  are  open  to  all 
DISA  acquisition  professionals  and  contractors  and  to 
guests.  The  T&E  forums  have  featured  nationally 
recognized  speakers  from  the  testing  community.  The 
bimonthly  T&E  management  symposiums  focus  on 
specific  issues  related  to  DOD  test  processes,  proce¬ 
dures,  and  methodologies. 

Conclusion 

DISA  TE&C  organizations  are  well  prepared  to 
conduct  mission-focused  and  agile  testing  across  all  IT 
portfolios,  and  they  strive  to  continuously  improve 
efficiency  through  use  of  virtualization,  enhanced  test 
automation,  collaboration,  and  federation  across 
DOD’s  overall  distributed  joint  test  infrastructure. 
Through  conscious  investment  decisions,  sensible 
allocation  of  resources,  and  new,  innovative  strategies, 
DISA  TE&C  organizations  will  enable  rapid  acquisi¬ 
tion  of  enhanced  capabilities  for  the  warfighter.  □ 
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Laboratories  Consortium.  E-mail:  chris.watson@disa.mil 


484  ITEA  Journal 


